Papers with full model tuning
SharPT: Shared Latent Space Prompt Tuning (2023.findings-eacl)
Copied to clipboard
| Challenge: | Prompt tuning is an efficient method for adapting large language models, but it is difficult and expensive to identify the source task that provides optimal prompts. |
| Approach: | They propose to learn a shared latent space which captures a set of basis skills from a mixture of source tasks and then transfer them to target tasks. |
| Outcome: | The proposed method outperforms previous methods on NLI, sentence completion, QA, conference resolution, word sense disambiguation and on various model scales. |
Modular Monolingual Adaptation using Pretrained Language Models (2026.acl-industry)
Copied to clipboard
| Challenge: | Existing approaches to building monolingual models for low-resource languages require a full model tuning process. |
| Approach: | They propose a modular approach to build monolingual models for low-resource languages by finetuning the whole model on the target language. |
| Outcome: | The proposed model improves on natural language understanding tasks on Scottish Gaelic, Irish, and Quechua with Quechuan being a very low-resource language. |
FaLA: Fast Linear Adaptation for Replacing Backbone Models on Edge Devices (2023.findings-emnlp)
Copied to clipboard
| Challenge: | Current NLP models heavily rely on pre-trained models, such as BERT and RoBERTa. |
| Approach: | They propose a lightweight method for personalized NLP classification tasks post-backbone replacement using a personalized matrix calculated from documents corresponding to users' old and new backbones. |
| Outcome: | The proposed method achieves over 1000 times computation reduction in Flops for backpropagation and brings the user-specific initialization for personal matrix yielding significant performance boost compared with popular transfer learning methods. |
Multitask Pre-training of Modular Prompt for Chinese Few-Shot Learning (2023.acl-long)
Copied to clipboard
| Challenge: | Prompt tuning is a parameter-efficient approach to adapting pre-trained language models to downstream tasks. |
| Approach: | They propose to combine pre-trained modules with pre-trains to boost prompt tuning for few-shot learning. |
| Outcome: | The proposed model outperforms prompt tuning, full model tuning, and prior prompt pre-training methods in few-shot learning settings. |